Field of the Invention
The present invention relates to a technique for generating mixed reality image data by combining virtual image data to captured image data.
Description of the Related Art
In recent years, a mixed reality technique, i.e., an MR technique has been known as a technique for mixing a real world and a virtual world seamlessly in real time. As one type of the MR technique, there is known a technique that uses a video see-through head mounted display (HMD) to allow an HMD user to observe a mixed reality image. The mixed reality image is created by imaging an object that substantially coincides with an object observed from a line-of-sight position of the HMD user with use of a video camera and the like, and displaying this captured image data with computer graphics (CG) superimposed thereon.
An imaging unit mounted on the video see-through HMD captures observation image data in an external world that substantially coincides with the line-of-sight position of the HMD user. The imaging unit includes two pairs of an image sensor and an optical system for a right eye and a left eye for generating stereo image data, and a digital signal processor (DSP) for performing image processing. A display unit displays the mixed reality image data created by combining CG to the object that substantially coincides with the object observed from the line-of-sight position of the HMD user. Further, the display unit is configured to deal with a pair of images of the right side and the left side like the imaging unit, and includes two pairs of a display device and an optical system for the right eye and the left eye.
A relationship between the external world and the mixed reality image data will be described now. The above-described display unit is configured to display image data input from an external apparatus without making any adjustment thereto. Therefore, even when the external world is at dust, the external world is at noon so that it is extremely bright, or the external world is under a slightly dark environment in the shade of a tree, generally, the mixed reality image data displayed on the display unit is image data unaffected by the environment of the external world around the HMD user, in which a brightness and a color are uniformly adjusted. Therefore, a gap may be generated between brightness and color sensations felt by the HMD user to the ambient environment, and the actual environment of the external world.
This influence cannot be also ignored for an optical see-through HMD. When CG data is superimposed on a see-through image of the external world in the optical see-through HMD, the brightness and the color are uniformly adjusted only in the CG data, whereby the balance and the color are unbalanced between the CG data and the see-through image of the external world. This leads to such a problem that the CG data may have an unnatural brightness and color to the see-through image of the external world depending on the environment of the external world, thereby impairing a realistic sensation.
Under this circumstance, Japanese Patent Application Laid-Open No. 2002-244077 discusses a technique that reduces a shutter speed of an imaging unit and increases a lighting time of an illumination light source of a display unit under a bright environment, while increasing the shutter speed and reducing the lighting time of the illumination light source under a dark environment. With this adjustment, the technique discussed in Japanese Patent Application Laid-Open No. 2002-244077 aims at maintaining the display brightness of the display unit at a brightness that matches see-through light even under various kinds of environments of an external world. Further, the technique discussed in Japanese Patent Application Laid-Open No. 2002-244077 aims at imaging an object while maintaining the brightness within a certain range by changing the shutter speed of the imaging unit according to the brightness of the external world.
Japanese Patent No. 03423402 discusses a technique that detects a color temperature of an external world by a sensor, and adjusts a color balance of display image data to be displayed on a display unit according to the detected color temperature. With this adjustment, the technique discussed in Japanese Patent No. 03423402 aims at generating display image data less unbalanced with see-through light.
However, the techniques discussed in Japanese Patent Application Laid-Open No. 2002-244077 and Japanese Patent No. 03423402 are techniques for adjusting the brightness and color of the display image data according to an illumination environment of the external world, and cannot display mixed reality image data that matches the illumination environment of the external world on the video see-through HMD.